v32.0.0

Compare Source

Breaking changes:

• fix rest api aiohttp timeout (#​337, @​soamicharan)

  This fix may affect you if you don't configure timeout for watch/stream. Previously it was treated as 5 min, now it's forever.

API Change

• ACTION REQUIRED for custom scheduler plugin developers:
  PodEligibleToPreemptOthers in the preemption interface now includes ctx in the parameters.
  Please update your plugins' implementation accordingly. (#​126465, @​googs1025) [SIG Scheduling]

• Changed NodeToStatusMap from a map to a struct and exposed methods to access the entries. Added absentNodesStatus, which informs the status of nodes that are absent in the map. For developers of out-of-tree PostFilter plugins, ensure to update the usage of NodeToStatusMap. Additionally, NodeToStatusMap should eventually be renamed to NodeToStatusReader. (#​126022, @​macsko) [SIG Node, Scheduling, and Testing]

• A new /resize subresource was added to request pod resource resizing. Update your k8s client code to utilize the /resize subresource for Pod resizing operations. (#​128266, @​AnishShah) [SIG API Machinery, Apps, Node and Testing]

• A new feature that allows unsafe deletion of corrupt resources has been added, it is disabled by default,
  and it can be enabled by setting the option --feature-gates=AllowUnsafeMalformedObjectDeletion=true.
  It comes with an API change, a new delete option ignoreStoreReadErrorWithClusterBreakingPotential has
  been introduced, it is not set by default, this maintains backward compatibility.
  In order to perform an unsafe deletion of a corrupt resource, the user must enable the option for the delete
  request. A resource is considered corrupt if it can not be successfully retrieved from the storage due to
  a) transformation error e.g. decryption failure, or b) the object failed to decode. Normal deletion flow is
  attempted first, and if it fails with a corrupt resource error then it triggers unsafe delete.
  In addition, when this feature is enabled, the 'details' field of 'Status' from the LIST response
  includes information that identifies the corrupt object(s).
  NOTE: unsafe deletion ignores finalizer constraints, and skips precondition checks.
  WARNING: this may break the workload associated with the resource being unsafe-deleted, if it relies on
  the normal deletion flow, so cluster breaking consequences apply. (#​127513, @​tkashem) [SIG API Machinery, Etcd, Node and Testing]

• Added singleProcessOOMKill flag to the kubelet configuration. Setting that to true enable single process OOM killing in cgroups v2. In this mode, if a single process is OOM killed within a container, the remaining processes will not be OOM killed. (#​126096, @​utam0k) [SIG API Machinery, Node, Testing and Windows]

• Added a /flagz endpoint for kube-apiserver endpoint. (#​127581, @​richabanker) [SIG API Machinery, Architecture, Auth and Instrumentation]

• Added a Stream field to PodLogOptions, which allows clients to request certain log stream (stdout or stderr) of the container.
  Please also note that the combination of a specific Stream and TailLines is not supported. (#​127360, @​knight42) [SIG API Machinery, Apps, Architecture, Node, Release and Testing]

• Added alpha support for asynchronous Pod preemption.
  When the SchedulerAsyncPreemption feature gate is enabled, the scheduler now runs API calls to trigger preemptions asynchronously for better performance. (#​128170, @​sanposhiho) [SIG Scheduling and Testing]

• Added driver-owned fields in ResourceClaim.Status to report device status data for each allocated device. (#​128240, @​LionelJouin) [SIG API Machinery, Network, Node and Testing]

• Added enforcement of an upper cost bound for DRA evaluations of CEL. The API server and scheduler now enforce an upper bound on the cost and runtime steps required for evaluating a CEL expression. (#​128101, @​pohly) [SIG API Machinery and Node]

• Added the ability to change the maximum backoff delay accrued between container restarts for a node for containers in CrashLoopBackOff. To set this for a node, turn on the feature gate KubeletCrashLoopBackoffMax and set the CrashLoopBackOff.MaxContainerRestartPeriod field between "1s" and "300s" in your kubelet config file. (#​128374, @​lauralorenz) [SIG API Machinery and Node]

• Allow for Pod search domains to be a single dot . or contain an underscore _ (#​127167, @​adrianmoisey) [SIG Apps, Network and Testing]

• Annotation batch.kubernetes.io/cronjob-scheduled-timestamp added to Job objects scheduled from CronJobs is promoted to stable. (#​128336, @​soltysh)

• Apply fsGroup policy for ReadWriteOncePod volumes. (#​128244, @​gnufied) [SIG Storage and Testing]

• Changed the Pod API to support resources at spec level for pod-level resources. (#​128407, @​ndixita) [SIG API Machinery, Apps, CLI, Cluster Lifecycle, Node, Release, Scheduling and Testing]

• ContainerStatus.AllocatedResources is now guarded by a separate feature gate, InPlacePodVerticalSaclingAllocatedStatus (#​128377, @​tallclair) [SIG API Machinery, CLI, Node, Scheduling and Testing]

• Coordination.v1alpha1 API is dropped and replaced with coordination.v1alpha2. Old coordination.v1alpha1 types must be deleted before upgrade (#​127857, @​Jefftree) [SIG API Machinery, Etcd, Scheduling and Testing]

• DRA: Restricted the length of opaque device configuration parameters. At admission time, Kubernetes enforces a 10KiB size limit. (#​128601, @​pohly) [SIG API Machinery, Apps, Auth, Etcd, Node, Scheduling and Testing]

• DRA: scheduling pods is up to 16x faster, depending on the scenario. Scheduling throughput depends a lot on cluster utilization. It is higher for lightly loaded clusters with free resources and gets lower when the cluster utilization increases. (#​127277, @​pohly) [SIG API Machinery, Apps, Architecture, Auth, Etcd, Instrumentation, Node, Scheduling and Testing]

• DRA: the DeviceRequestAllocationResult struct now has an "AdminAccess" field which should be used instead of the corresponding field in the DeviceRequest field when dealing with an allocation. If a device is only allocated for admin access, allocating it again for normal usage is now supported, as originally intended. To allow admin access, starting with 1.32 the DRAAdminAccess feature gate must be enabled. (#​127266, @​pohly) [SIG API Machinery, Apps, Auth, Etcd, Network, Node, Scheduling and Testing]

• Disallow k8s.io and kubernetes.io namespaced extra key in structured authentication configuration. (#​126553, @​aramase) [SIG Auth]

• Fixed a bug in the NestedNumberAsFloat64 Unstructured field accessor that could have caused it to return rounded float64 values instead of errors when accessing very large int64 values. (#​128099, @​benluddy)

• Fixed the bug where spec.terminationGracePeriodSeconds of the pod will always be overwritten by the MaxPodGracePeriodSeconds of the soft eviction, you can enable the AllowOverwriteTerminationGracePeriodSeconds feature gate, which will restore the previous behavior. If you do need to set this, please file an issue with the Kubernetes project to help contributors understand why you needed it. (#​122890, @​HirazawaUi) [SIG API Machinery, Architecture, Node and Testing]

• Graduated Job's ManagedBy field to beta. (#​127402, @​mimowo) [SIG API Machinery, Apps and Testing]

• Implemented a new, alpha seLinuxChangePolicy field within a Pod-level securityContext, under SELinuxChangePolicy feature gate. This field allows for opting out from mounting Pod volumes with SELinux label when SELinuxMount feature is enabled (it is alpha and disabled by default now).
  Please see the KEP how we expect to warn users before any SELinux behavior changes and how they can opt-out before. Note that this field and feature gate is useful only with clusters that run with SELinux enabled. No action is required on clusters without SELinux. (#​127981, @​jsafrane) [SIG API Machinery, Apps, Architecture, Node, Storage and Testing]

• Introduced v1alpha1 API for mutating admission policies, enabling extensible # admission control via CEL expressions (KEP 3962: Mutating Admission Policies). # To use, enable the MutatingAdmissionPolicy feature gate and the admissionregistration.k8s.io/v1alpha1 # API via --runtime-config. (#​127134, @​jpbetz) [SIG API Machinery, Auth, Etcd and Testing]

• Introduced compressible resource setting on system reserved and kube reserved slices. (#​125982, @​harche)

• kube-apiserver: Promoted the StructuredAuthorizationConfiguration feature gate to GA. The --authorization-config flag now accepts AuthorizationConfiguration in version apiserver.config.k8s.io/v1 (with no changes from apiserver.config.k8s.io/v1beta1). (#​128172, @​liggitt) [SIG API Machinery, Auth and Testing]

• kube-proxy now reconciles Service/Endpoint changes with conntrack table and cleans up only stale UDP flow entries (#​127318, @​aroradaman) [SIG Network and Windows]

• kube-scheduler removed AzureDiskLimits ,CinderLimits EBSLimits and GCEPDLimits plugin. Given the corresponding CSI driver reports how many volumes a node can handle in NodeGetInfoResponse, the kubelet stores this limit in CSINode and the scheduler then knows the limit of the driver on the node. Removed plugins AzureDiskLimits, CinderLimits, EBSLimits and GCEPDLimits if you explicitly enabled them in the scheduler config. (#​124003, @​carlory) [SIG Scheduling, Storage and Testing]

• kubelet: the --image-credential-provider-config file was loaded with strict deserialization, which failed if the config file contained duplicate or unknown fields. This protected against accidentally running with malformed config files, unindented files, or typos in field names, and it prevented unexpected behavior. (#​128062, @​aramase) [SIG Auth and Node]

• NodeRestriction admission now validates the audience value that kubelet is requesting a service account token for is part of the pod spec volume. This change is introduced with a new kube-apiserver featuregate ServiceAccountNodeAudienceRestriction that's enabled by default. (#​128077, @​aramase) [SIG Auth, Storage and Testing]

• Promoted CustomResourceFieldSelectors to stable; the feature was enabled by default. The --feature-gates=CustomResourceFieldSelectors=true flag was no longer needed on kube-apiserver binaries and would be removed in a future release. (#​127673, @​jpbetz) [SIG API Machinery and Testing]

• Promoted feature gate StatefulSetAutoDeletePVC from beta to stable. (#​128247, @​mattcary) [SIG API Machinery, Apps, Auth and Testing]

• Removed all support for classic dynamic resource allocation (DRA). The DRAControlPlaneController feature gate, formerly alpha, is no longer available. Kubernetes now only uses the structured parameters model (also alpha) for allocating dynamic resources to Pods.

  if and only if classic DRA was enabled in a cluster, remove all workloads (pods, app deployments, etc. ) which depend on classic DRA and make sure that all PodSchedulingContext resources are gone before upgrading. PodSchedulingContext resources cannot be removed through the apiserver after an upgrade and workloads would not work properly. (#​128003, @​pohly) [SIG API Machinery, Apps, Auth, Etcd, Node, Scheduling and Testing]

• Removed generally available feature gate HPAContainerMetrics (#​126862, @​carlory) [SIG API Machinery, Apps and Autoscaling]

• Removed restrictions on subresource flag in kubectl commands (#​128296, @​AnishShah) [SIG CLI]

• Revised the kubelet API Authorization with new subresources, that allow finer-grained authorization checks and access control for kubelet endpoints.
  Provided you enable the KubeletFineGrainedAuthz feature gate, you can access kubelet's /healthz endpoint by granting the caller nodes/helathz permission in RBAC.
  Similarly you can also access kubelet's /pods endpoint to fetch a list of Pods bound to that node by granting the caller nodes/pods permission in RBAC.
  Similarly you can also access kubelet's /configz endpoint to fetch kubelet's configuration by granting the caller nodes/configz permission in RBAC.
  You can still access kubelet's /healthz, /pods and /configz by granting the caller nodes/proxy permission in RBAC but that also grants the caller permissions to exec, run and attach to containers on the nodes and doing so does not follow the least privilege principle. Granting callers more permissions than they need can give attackers an opportunity to escalate privileges. (#​126347, @​vinayakankugoyal) [SIG API Machinery, Auth, Cluster Lifecycle and Node]

• The core functionality of Dynamic Resource Allocation (DRA) got promoted to beta. No action is required when upgrading, the previous v1alpha3 API is still supported, so existing deployments and DRA drivers based on v1alpha3 continue to work. Downgrading from 1.32 to 1.31 with DRA resources in the cluster (resourceclaims, resourceclaimtemplates, deviceclasses, resourceslices) is not supported because the new v1beta1 is used as storage version and not readable by 1.31. (#​127511, @​pohly) [SIG API Machinery, Apps, Auth, Etcd, Node, Scheduling and Testing]

• The default value for node-monitor-grace-period has been increased to 50s (earlier 40s) (Ref - https://github.com/kubernetes/kubernetes/issues/121793) (#​126287, @​devppratik) [SIG API Machinery, Apps and Node]

• The resource/v1alpha3.ResourceSliceList filed which should have been named "metadata" but was instead named "listMeta" is now properly "metadata". (#​126749, @​thockin) [SIG API Machinery]

• The synthetic "Bookmark" event for the watch stream requests will now include a new annotation: kubernetes.io/initial-events-list-blueprint. THe annotation contains an empty, versioned list that is encoded in the requested format (such as protobuf, JSON, or CBOR), then base64-encoded and stored as a string. (#​127587, @​p0lyn0mial) [SIG API Machinery]

• To enhance usability and developer experience, CRD validation rules now support direct use of (CEL) reserved keywords as field names in object validation expressions.
  Name format CEL library is supported in new expressions. (#​126977, @​aaron-prindle) [SIG API Machinery, Architecture, Auth, Etcd, Instrumentation, Release, Scheduling and Testing]

• Updated incorrect description of persistentVolumeClaimRetentionPolicy (#​126545, @​yangjunmyfm192085) [SIG API Machinery, Apps and CLI]

• X.509 client certificate authentication to the kube-apiserver now produces credential IDs (derived from the certificate's signature) , for use in audit logging. (#​125634, @​ahmedtd) [SIG API Machinery, Auth and Testing]

v31.1.1

Compare Source

Changes:

• Allow resource names >2 parts (#​343, @​edwinpjacques)

v31.1.0

Compare Source

Breaking changes:

• Websocket connect method returns an asynchronous context manager instead of a websocket (#​328, @​olivier-matz-6wind)

  Example:

  websocket = await core_v1_ws.connect_get_namespaced_pod_exec(...)